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ABSTRACT 


The Department of Defense has mandated the integration of digital engineering 
into acquisition engineering throughout the entire life cycle. One critical element to 
achieve this directive is the incorporation of Model-Based Systems Engineering (MBSE) 
into System Engineering Technical Reviews (SETR). However, little research exists that 
provides an evaluation of the current MBSE environment’s ability to satisfy SETR 
criteria. Furthermore, no research exists that assesses the state of SETR criteria with 
respect to its readiness for MBSE integration. This thesis performs this missing research 
by creating a process to assess SETR criteria. The process was then employed to assess 
Preliminary Design Review (PDR) criteria with respect to their suitability, quality, and 
readiness for integration into the current MBSE environment. The process is agnostic to 
the SETR event and the criterion’s background. A verification of previous findings 
related to the current MBSE environment’s ability to satisfy SETR criteria was 
performed. The research found that the PDR criteria is unable to integrate with the 
current MBSE environment, requiring changes to enable integration, including new 
MBSE views. This finding verifies the previous findings. The final finding notes that 
regardless of MBSE integration, the PDR criteria require significant curation in order to 
provide the necessary robustness and effectiveness to provide insight into the health and 


status of a Sol under review. 


THIS PAGE INTENTIONALLY LEFT BLANK 


vi 


Il. 


III. 


PN FRODUCTION wsvades sans csocusutesesdoavesenvevees sansteseecusetcuis socsucucueddessseondunseapetensitens 
A. BACKGROUND ssid naan nies 
B. CB VEE CTY EG sosieiis sca scsss cc eetisateaaeniaies vanish deni natesvatceateacsuoatneieaceiccecneenas 
C. THE RESEARCH QUESTIONS ..0........cccscccsscsssccssscssceccssscsecsscsesces 
D. RESEARCH METHOD sssevcscccsevccsesssessovisve cascescedeesesvcsvesteevecccsesesssohans 
E. SCOPE, LIMITATIONS, AND ASSUMPTIONS ............sccssssssssseeee 
F. ORGANIZATION OF STUDY .........sssscsssccssssscsssssessescsesssscsescsesseesees 
LITERATURE, REVIEW scsscutcecsdesvesescsseussosseeceecusstcacs secs escodssvesssvensesoneveteosstee 
A. BACK GROUND scien etic a leeaees 
B. REVIEW OF EVOLVING MODEL-BASED SYSTEMS 
ENGINEERING ONTOLOGIES AND STRUCTUREG.................. 
C. REVIEW OF DEPARTMENT OF DEFENSE DIGITAL 
ENGINEERING STRATEGY ..........cccccssssssssscssccssscsecsscssscsecssscssseeee 
D. REVIEW OF DEFENSE ACQUISITION GUIDEBOOK: 
CHAPTER 3 ossecpiscsstencesiecevesiesseacivithe eves tosdestbcsvindectendavetbeuiious tote cttonces 
E. REVIEW OF A ROADMAP FOR ADVANCING THE STATE 
OF THE PRACTICE OF MBSE FOR GOVERNMENT 
ACQUISITION si scccsses se sscsvccsccsowssvecsacveceseveusecenadudecsteensvccicendeesesesteexens 
F. REVIEW OF DEFINING A MODEL-BASED SYSTEMS 
ENGINEERING APPROACH FOR MILESTONE 
FECHNIGAE, RE VUE WSS cosviccseditctesucash coseciacasaces sceutetacerees feoiteoesdeiecices 
G. SUMMARY OF LITERATURE REVIEW. .............ccscssscssssessssssseeees 
PIROGHSS scoesvecaesh tea taliescaascacste tae cecaute este satiaceeneace cette saatheseetleaneeeen remem tess 
A. DUR ROSE oegsccese5etctee co tacechic areas cesta cats ccktecncenect tae, etecais 
B. POPP OAC i scscscctets letideede as lovsadadatoctusedavcalesaeus cust lust sdecadedewwatevesastocdeanants 
1. Review, Document, and Alter SETR Criteria ...............scese0 


TABLE OF CONTENTS 


2. Assess and Document Each Current SETR Event 


Criterion’s Level of Development................scccsssccsssscessssceseeces 


3. Assess and Document Each SETR Criterion’s Ability to 


Provide Insight into the Health and Status of the Sol.......... 


4. Assess and Document SETR Criteria Ability to be 


Mapped to an Existing MBSE View.............sccssscssssssssssscesees 


5. Consolidate Documentation of All Assessed SETR 


ere 
CY UACT IA sisdibecescicccescdssidesdcsocsesncdscedeecdecsoscse doaesdcvescveacesdasedévecsoascse 


Vii 


PVs: DAAC ANA YSIS ses sussecencccpetandesosusects staiesavacusesnied sactietsuscddvcdetesaoteusecuiedvadeledvives 33 


A. ISSUE RESEARCHED AND DATA ANALYZED.........ccscsssssssssesseees 33 
B. RESEA ROH RESULTS wsissessaassvisssuesicacaiacvtteseeucoatencavassexessaccedcceateevensstes 34 
1. Results of Function 1.0: Assessment of Characteristic 1.......... 34 
Zs Results of Function 2.0: Assessment of Characteristic 2.......... 38 
3. Results of Function 3.0: Assessment of Characteristic 3.......... 42 
C. INTERPRETATIONS OF RESULTS ...........ccscssssssssscssscssscsscsesscseescees 44 
V. CONCTCUSIONS, ca cuciesessoaslsvenesvsaesssverdesosvevescnsaesaveceuvteasutveasconseosscsasveracconstsasvteuseedl 47 
A. SS WIVEINTARY ciiendis cileesasvcnseceneceichidanetneaad chedia ove vesseeeekedonatictneiee av ateeieey 47 
B. RECOMMENDATIONS FOR FUTURE WORK AND 
CONSIDERATION ‘cscissscscecosecescatevteesednossasieanesvecssensseutavecesebsssoncioasoscesnens 48 


SUPPLEMENTAL: NAVAIR PDR CRITERIA AND RESULTS OF 


ASSESSIMEIN D sesssscicssshsizessisestedwanicocsssvetecaseuascasbacesecangnsoasesthoscessecadeessadeaocssaneceassed 51 
LIST OP REFERENCES 'cscsiies tess sdeccedsasascatwidectesteiesdascdccsseseussctasssiesdessstetdocecdsescdesiuseested aie) 
INETIAL- DISTRIBU TION DUST csssscascovecsseteissuaiss vovsneceastessceusususteistessesvorbdatwasusasceviensace BE) 


Vill 


Figure 1. 


Figure 2. 


Figure 3. 


Figure 4. 


Figure 5. 


Figure 6. 


Figure 7. 


Figure 8. 


Figure 9. 


Figure 10. 


Figure 11. 


Figure 12. 


LIST OF FIGURES 


Review, Document, and Alter SETR Criteria LML Action Diagram ....... 


Assess and Document Each Current SETR Event Criterion’s Level 
of Development LML Action Diagram .00...... eee eeeeeeeeeereeeneeeeneeeeeeeeneees 


Assess and Document Each Partially Consistent SETR Criterion 
Ability to be Reworded to Allow Consistency with Level of 
Development for Current SETR Event LML Action Diagram................... 


Assess and Document Each SETR Criterion’s Ability to Provide 
Insight into the Health & Status of the Sol LML Action Diagram............ 


Assess and Document SETR Criteria with Partial Ability to Provide 
Insight into the Health and Status of the Sol if Reworded LML 
PunetiGn Diagram ics siisasdedanicetsssacsasvacsaacerenséan ds sescgatasusnacsoanddceveneadaacndasactase 


Assess and Document SETR Criteria Ability to be Mapped to an 
Existing MBSE View LML Action Diagram ......... cee eee eeneceeeeeeeeeeeeees 


Consolidate Documentation of All Assessed SETR Criteria LML 
PFCUOM DIAGN yss ities casero eed ese sas census Sessea edu aa Pana esaadedgtace tucson eee 


Distribution of Assessed Criteria Regarding Their Consistency to an 
Expected Level of Development for an Sol during the Preliminary 
MOGSTS 1, PASE xy ccautccasisrstarccatesaeecasacaaaes)saauascecl enucateterseehad beceansasuueetieneuue seas 


Distribution of Assessed Criteria Regarding Their Ability to Provide 
Insight into the Health and Status of the Solo... ee eee esse ceeeeeeeeeeneees 


Distribution of Assessment Results of Criteria with Partial Ability to 
Provide Insight Regarding Their Ability for Rewording to Bestow 
the Ability to Provide Insight 4s. ccsacssversatvsteeacen thevcdeshensnceesenceedtansvenedoorateene 


Distribution of Assessed Criteria Regarding Their Ability to map to 
al EXISUNS WISE, VICW. co:.25cccts cisscseacorcactas eacase eee eaten aaa aS: 


Distribution of Remaining Criteria After the First Two Assessments 
Detailing the Characteristics Requiring Rewording .............:escceeeseeeeeteeees 


1X 


THIS PAGE INTENTIONALLY LEFT BLANK 


Table 1. 


Table 2. 


Table 3. 


Table 4. 


LIST OF TABLES 


DOD Acquisition Technical Maturity Points. Adapted from Defense 
Acquistion University (2017) 43-47). scccteccssssneassteonied padi endicenne 


Examples of Criteria as Written Contrasted with Suggested 
REWOTdING v.pdecsicesieapaitessharisisee desi deed aon leuann 


Examples of Criteria as Written Contrasted with Possible Rewording..... 


Distribution of Criteria within Most Promising Thematic Areas .............. 


Xl 


Pe) 


THIS PAGE INTENTIONALLY LEFT BLANK 


xii 


LIST OF ACRONYMS AND ABBREVIATIONS 


ASR Alternative Systems Review 

CDD Capability Development Document 
CDR Critical Design Review 

DAG Defense Acquisition Guidebook 

DAU Defense Acquisition University 

DE Digital Engineering 

DOD Department of Defense 

DoDAF Department of Defense Architecture Framework 
FCA Functional Configuration Audit 
FDDR Full Deployment Decision Review 
FRP DR Full-Rate Production Decision Review 
GAO Government Accountability Office 
LML Lifecycle Management Language 
MBSE Model-Based Systems Engineering 
MSA Material Solution Analysis 

NAVAIR Naval Air System Command 

OT&E Operational Test & Evaluation 

PCA Physical Configuration Audit 

PDR Preliminary Design Review 

PRR Production Readiness Review 

SETR Systems Engineering Technical Review 
SFR System Functional Review 

Sol System of Interest 

SRR Systems Requirements Review 

SVR System Verification Review 

SysML Systems Modeling Language 


UML Unified Modeling Language 


Xlil 


THIS PAGE INTENTIONALLY LEFT BLANK 


XIV 


EXECUTIVE SUMMARY 


The Department of Defense’s (DOD) digital engineering Strategy defines five goals 
and the vision of integrating digital engineering (DE) into DOD systems throughout their 
full life cycles. Goal | specifies establishing formal methods to support the development, 
integration, and the universal employment of digital models as a fundamental element of 
engineering activities to provide a comprehensive digital representation of the system of 
interest (Sol) throughout its entire life cycle. Goal 1 is further broken down into three 
separate, but interrelated, sub-goals: formalize the planning for models to support 
engineering activities and decision making across the life cycle; formally develop, 
integrate, and curate models; and use models to support engineering activities and decision 
making across the life cycle (Office of the Deputy Assistant Secretary of Defense for 
Systems Engineering 2018). 


Model-based systems engineering (MBSE) provides a method for performing 
systems engineering in a model-based environment. Although, MBSE does not encompass 
all the various disciplines, or their models, it does capture a significant portion of a system’s 
information necessary to model an Sol as directed within the DOD digital engineering 
Strategy. In support of executing Goal 1 of the DOD digital engineering Strategy, a 
significant need exists to generate a process to curate and thereby improve the existing 


System Engineering Technical Review (SETR) criteria for use within MBSE. 


According to the findings by Vaneman, Carlson, and Wolfgeher (2020), numerous 
Preliminary Design Review (PDR) criteria are not suitable to be addressed within a MBSE 
environment. The findings also stated that a significant portion of the PDR criteria 
shortcomings related to their binary nature. This binary nature instills within the criteria a 
lack of robustness, thereby limiting their usefulness in evaluating the Sol in general, 


regardless of the employment of MBSE. 


This thesis will address the binary nature of the criteria along with other faults that 
prevent the criteria from providing insight into the Sol and limit the benefits of MBSE. 


Performing an assessment of those PDR criteria shall take place through the employment 


XV 


of the process that will assess, document, and alter (as appropriate) criteria that lack the 


required robustness and usefulness. 


The scope of the thesis is to define the establishment of a process that details how 
to curate existing SETR criteria, and alter them as necessary, to more accurately capture 
the “health and status” of a Sol under review in preparation for integration of model-based 
views and data. The process created shall guide users through gated questions that shall 
assist in the curation of the existing SETR criteria by categorizing, documenting, and if 
necessary altering or removing them. The thesis shall employ this process to perform a 
review of previously gathered and assessed PRD criteria to verify the need for the creation 
of the described process and to verify the previous findings. This review shall justify the 
creation of such a process by verifying the findings of the PRD criteria previously 


evaluated. 


One core deliverable of the thesis is a process, employable within a formalized plan, 
which enables the evaluation of SETR criteria. The evaluation consists of three separate 
assessments of each SETR criterion for three specific characteristics that each should 
possess. The first characteristic assessed is the consistency with, or appropriateness to, the 
required level of development addressed by the SETR criterion in comparison to the current 
SETR event. The second characteristic is the SETR criterion’s quality to assess or its ability 
to provide insight into the health and status of the program or Sol under review. The final 
characteristic is each SETR criterion’s ability for representation by an existing MBSE 
view. These assessments will determine a set of SETR criteria ability to evaluate an Sol, 
to what extent criteria can map to existing MBSE views and the scope of criteria that 
require additional alteration either to the criteria or to MBSE views to permit the mapping 


of the criteria. 


The employment of this process as the companying research for this thesis occurred 
on data that solely consisted of the NAVAIR PDR criteria. The supplemental Microsoft 
Excel document captures the NAVAIR PDR criteria analyzed and the results of the 


research. 
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When performing the criteria assessments, the research did not distinguish between 
the criteria area of focus (e.g., programmatic or technical). The research only assessed each 


criterion based on the three previously mentioned characteristics. 


The results of the research performed indicate not only are most of the reviewed 
PDR criteria not ready for MBSE integration, but they require curation for employment in 


the current intended usage. 


The data shows that over 43% of the criteria deemed partially or fully inconsistent 
with the level of development at preliminary design and nearly all criteria evaluated as 
binary in nature. To provide perspective on the prevalence of binary criteria, over 95% of 
the criteria that had their Characteristic 2 assessed contained a binary nature. Taking a 
deeper look at the data after the first two assessments of the remaining criteria, as the 
removal of some criteria occurred; an observation is possible along with possible 
implications. An observation from this deeper look is that without rewording only 2.54% 
of all PDR criteria assessed are consistent with the required or expected level of 
development and have the ability to provide insight into the health and status of the Sol. 
The possible implication is all SETR criteria likely require some level of curation unless 


the reviewed PDR criteria are severe outliers. 


In summarizing the results, the execution of an assessment of the SETR’s criteria 
critical characteristics provided insight into the health and status of the criteria. This insight 
supplied the knowledge of whether the SETR criteria would have the ability to accomplish 
their intended purpose. The results of the assessment presented a bleak reality of the SETR 
criteria reviewed. Over 43% of the criteria did not fully address the appropriate level of 
development as aligned with the SETR event. The criteria that proceeded to the next 
assessment had nearly all criteria, approximately 97%, partially or fully failing to meet the 
evaluation requirement of having the ability to provide insight into the health and status of 
the Sol. The final assessment of the remaining criteria found that only approximately 13% 


of the criteria assessed had the ability to map partially of fully to an existing MBSE view. 
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A conclusion one can draw from the data is the effectiveness of any PDR performed 
with this set of criteria lacks the intended robustness and usefulness when attempting to 


provide insight into the health and status of the Sol under review. 


Four recommendations for future work or consideration originate from the thesis 
findings. The first recommendation is that additional SETR criteria undergo assessment 
and curation in preparation for MBSE integration. Having healthy and robust set of SETR 
criteria that can accurately capture the “health and status” of an Sol under review is 


necessary prior to the integration and implementation of MBSE into SETR events. 


The second recommendation is for the acquisition community to assign necessary 
attributes to SETR criteria during their development and curation. The performance of this 
effort would mirror what Systems Engineering teaches its practitioners regarding the 


assignment of necessary attributes for requirements during their development. 


The third recommendation is to investigate the creation of additional MBSE views 
that will support the evaluation of Sols. A need exists to perform an evaluation regarding 


the needed MBSE views to support a curated set of SETR criteria. 


The final recommendation is to consider the automation of the applicable portions 
of SETR criteria assessments. The employment of Formal Methods in conjunction with the 
development of a software program could directly handle some evaluations while other 
assessments could have their burden reduced through the assistance the software provides 


by decreasing the initial workload prior to human interaction of assessing SETR criteria. 
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I. INTRODUCTION 


A. BACKGROUND 


The Department of Defense’s (DOD) digital engineering Strategy defines five goals 
and the vision of integrating digital engineering (DE) into DOD systems throughout their 
full life cycles. Goal | specifies establishing formal methods to support the development, 
integration, and the universal employment of digital models as a fundamental element of 
engineering activities to provide a comprehensive digital representation of the system of 
interest (Sol) throughout its entire life cycle. The formal methods will afford the ability to 
maintain consistent performance of analysis and decision making for programs throughout 
the DOD enterprise. Goal | is further broken down into three separate, but interrelated, 
sub-goals: formalize the planning for models to support engineering activities and decision 
making across the life cycle; formally develop, integrate, and curate models; and use 
models to support engineering activities and decision making across the life cycle (Office 


of the Deputy Assistant Secretary of Defense for Systems Engineering 2018). 


Model-based systems engineering (MBSE) provides a method for performing 
systems engineering in a model-based environment. MBSE does not encompass all of the 
various disciplines, or their models, but it does capture a significant portion of a system’s 
information necessary to model an Sol as directed within the DOD digital engineering 
Strategy. However, presently a disconnect exists between a major element in DOD 
acquisition, System Engineering Technical Reviews (SETR), and MBSE. In support of 
executing Goal 1 of the DOD digital engineering Strategy, a significant need exists to 
generate a process to curate and thereby improve the existing System Engineering 
Technical Review (SETR) criteria for use within MBSE. This process would afford the 
ability to fully integrate and permit the utilization of the content required to fulfill the 


SETR’s criteria throughout the acquisition life cycle. 


B. OBJECTIVES 


According to the findings by Vaneman, Carlson, and Wolfgeher (2020), numerous 
Naval Air System Command (NAVAIR) Preliminary Design Review (PDR) criteria are 
1 


not suitable to be addressed within the current MBSE environment. The findings also stated 
that a significant portion of the PDR criteria shortcomings related to their binary nature. 
This binary nature instills within the criteria a lack of robustness, thereby limiting their 
usefulness in evaluating the Sol in general, regardless of the employment of MBSE. The 
creation of a process to curate the SETR criteria and alter them as necessary will allow a 
robust inquiry and thereby greater insight into the Sol under review. This additional insight 


will permit a more accurate understanding of the “health and status” of the Sol. 


C; THE RESEARCH QUESTIONS 


1. How does one determine if a process to curate existing SETR criteria and 
alter them as necessary to more accurately capture the “health and status” 


of an Sol under review through model-based views and data is required? 


2 How does one create a process to curate existing SETR criteria and alter 
them as necessary to more accurately capture the “health and status” of an 


Sol under review through model-based views and data? 


3. If required, how does one verify that a process generated to address the 


problem fulfills its intended purpose? 


D. RESEARCH METHOD 


The thesis will address one of the major faults of the NAVAIR PDR criteria as 
Vaneman, Carlson, and Wolfgeher’s paper (2020) asserts. Specifically, that a lack of 
robustness prevents the criteria from providing insight into the Sol and limits the benefits 
of integrating MBSE. To address this topic the creation of a process shall occur. The 
process will assess, document, and alter (as appropriate) SETR criteria that prevent or limit 
a thoughtful examination of an Sol. Following the creation of the process, an assessment 
of the previously stated PDR criteria shall take place through the employment of the 
process. The design and verification of this process acts as the core of the research method 


employed that is the basis of this thesis. 


E. SCOPE, LIMITATIONS, AND ASSUMPTIONS 


The scope of this thesis is to establish a process that details how to curate existing 
SETR criteria, and alter them as necessary, to more accurately capture the “health and 
status” of an Sol under review through model-based views and data. The process shall 
guide users through gated questions that assist in the curation of the existing SETR criteria 


by categorizing, documenting, and if necessary altering or removing them. 


This thesis includes a review of NAVAIR PRD criteria. This review will act as a 
verification of the process and provide an example demonstration of the process on set of 
existing PDR criteria. Additionally, the review will employ the process to evaluate the 


ability of the criteria to map to existing MBSE view. 


Finally, a validation of the necessity of the process will occur with the verification 
of the existence of disproportionate number of PDR criteria containing traits that prevent 


or limit a thoughtful examination of an Sol. 


Three specific limitations bound the scope of the thesis. The first limitation is the 
set of SETR criteria that the thesis will evaluate. Academic review and notational 
observance are the driving elements to limit the thesis’ scope to these criteria. Through 
observations of programs, projects, working groups, and communities of interest the 
current leading edge of MBSE employment within DOD acquisition is at SETRs just before 
or at the level PDR. The thesis shall cover the NAVAIR PDR criteria reviewed and 
addressed within Vaneman, Carlson, and Wolfgeher (2020) paper where they argue the 
current state MBSE is unable to fully-address all of the required criteria to comprehensively 
evaluate a given system during a PDR. Based on these elements the current largest need 
for improvement is at the PDR level, thereby establishing the bounding focus on PDR 


criteria exclusive of other SETR criteria. 


The second limitation is the extent of the process presented. The thesis shall define 
the establishment of a process that details how to curate existing SETR criteria, and alter 
them as necessary, to more accurately capture the “health and status” of an Sol under 
review in preparation for implementation of model-based views and data. However, the 


thesis will not address the creation of new MBSE views to permit the capture of any 
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criteria, and its data, not currently able to map to an existing MBSE view the criteria. This 


limitation bounds the scope to the existing PDR criteria and MBSE views. 


The final limitation on the scope of the thesis relates to the MBSE language 
employed. The thesis employs the Life cycle Management Language (LML) as part of the 
research and in the creation of the process. While the LML has the ability to map its 
functions to other MBSE languages, various documents have presented LML’s enhanced 
ability to model the full life cycle of an Sol in ways other MBSE languages currently lack 
(Vaneman, Sellers, and Dam 2018). 


The thesis makes the following two assumptions. First, the NAVAIR PDR criteria 
covers a sufficient breath of SETR criteria to make the thesis and the presented process 
agonistic regarding the various DOD branches. The second assumption is while not all 
readers may know or employ LML, the readers are able to translate the process into their 
language and tool of choice through the employment of language mapping that exist 


external to this thesis. 


F. ORGANIZATION OF STUDY 


This introductory chapter has provided a general outline of the thesis with special 
focus given to the background, objectives, scope, limitations, and assumptions. The 
subsequent chapters shall expand on select portions of this chapter. Chapter II shall detail 
the literature review performed. Chapter III shall present the purpose of the process created 
and define the approach of the process. Chapter IV shall describe the issue addressed along 
with the data analyzed, the results of the research, and finally provide an interpretation of 
the results. Chapter V shall offer a summary of the work performed, recommendations 


based on the research performed and its successive results, and detail potential future work. 


I. LITERATURE REVIEW 


A. BACKGROUND 


To date the majority of the literature regarding MBSE has focused on the benefits 
MBSE can bring to the field of systems engineering (Estefan 2008; Jackson, Wilkerson 
and Castet 2016; Holladay et al. 2019; Noguchi 2019) or benefits to a system throughout 
its entire life cycle (Vaneman, Sellers and Dam 2018; Goldwasser and Ryder 2019). In 
addition, some documents (Dam 2014) have addressed how MBSE can meet the 
requirements of the DOD Architecture Framework (Department of Defense 2009) in 
support of systems engineering and acquisition. Only recently has literature been published 
that begins to investigate the implementation of integrating MBSE into the initial phases 
of the DOD acquisition (Vaneman and Carlson 2019). To date, one of most complete 
literature based reviews of implementing the integration of MBSE primarily focused on 
SETRs up to and including PDR determined that a large majority of PDR criteria were 
unable to be sufficiently addressed by existing MBSE views. As part of the findings it was 
documented that a majority of the PDR criteria were binary in nature and thereby provided 
little to no insight into the “health and status” of the Sol. These PDR criteria, which are 
binary in nature, do not afford information or data that is suitable to robust employment 


within a MBSE environment (Vaneman, Carlson, and Wolfgeher 2020). 


B. REVIEW OF EVOLVING MODEL-BASED SYSTEMS 
ENGINEERING ONTOLOGIES AND STRUCTURES 

MBSE provides a method for performing systems engineering in a model-based 
environment that limits or removes the need for the generation of document-based artifacts. 
However, there exists numerous varying thoughts on different processes and methods of 
how to define MBSE. This fractured view of MBSE prevents a single cohesive 
implementation. Vaneman states: 

However, more than a decade after it was defined, MBSE remains a 

mysterious concept that means many things to many different people and 


organizations. Some will contend that MBSE is the Unified Modeling 
Language (UML) or Systems Modeling Language (SysML), while others 
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believe that MBSE is the Department of Defense Architecture Framework 
(DoDAF) or the Zachman Framework. As a result, MBSE is frequently 
implemented as a series of “architectural products,” instead of a singular 
virtual representation of the system that can be used for a wide array of 
systems engineering analysis. This due, in part, to lack of an adopted 
ontology that defines entities, and provides the structure, used to represent 
systems. (W. K. Vaneman 2018, 1) 


Vaneman argues that with an agreed upon singular formalized ontology, the field 
of systems engineering could begin the full implementation of MBSE permitting improved 
engineering support and management of system complexity. Vaneman presents the 
following definition of MBSE, “as the formalized application of modeling (static and 
dynamic) to support system design and analysis, throughout all phases of the system life 
cycle, through the collection of modeling languages, structures, model-based processes, 
and presentation frameworks used to support the discipline of systems engineering in a 
model-based or model-driven context” (Vaneman 2018, 2). Four components establish this 


definition of MBSE according to Vaneman, who denotes them as follows: 


|. Modeling Languages — Serves as the basis of tools and enable the 
development of system models. Modeling languages are based on a 
logical construct (visual representation) and/or an ontology. 

2. Structure — Uses the ontology, and defined relationships between the 
systems entities, to establish concordance, thus allowing for the 
emergence of system behaviors and performance characterizations 
within the model. 

3. Model-Based Processes — Provides the analytical framework to conduct 
the analysis of the system virtually defined in the model. The model- 
based processes may be traditional systems engineering processes such 
as requirements management, risk management, or analytical methods 
such as discrete event simulation, systems dynamics modeling, and 
dynamic programming. 

4. Presentation Frameworks - Provides the framework for the logical 
constructs of the system data in visualization models that are appropriate 
for the given stakeholders. These visualization models take the form of 
traditional systems engineering models. These individual models are 
often grouped into frameworks that provide the standard views and 
descriptions of the models, and the standard data structure of 
architecture models. (2018, 2) 


At the intersection of these four components lays the possibility for MBSE’s 


greatest potential efficacy. It is critical for the DOD and systems engineering to have a 
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single cohesive implementation of MBSE that provides its greatest potential efficacy. The 
scope and complexity of acquisition programs is ever increasing beyond the human 
capacity to comprehend it in its entirety at once. Even with the proper employment of 
systems engineering utilizing document-based work products, acquisition projects are out 
pacing engineers’ ability to agree to changes; document them; and implement them while 
providing insight into the Sol’s changes and all of the possible impacts in a timely manner 
to management and leadership. Through the implementation of MBSE, the possibility 
exists to provide a single continuous current understanding of the SoI where no lag time is 
present. When presenting information and data on the Sol the viewer can observe the 
current-status and not a status of when the collection of the information and data occurred. 
Additionally, there is only one source of data. This removes possible confusions of which 
version is current; providing a “single source of truth” for the Sol regarding its information 
and data. These features offer the potential to provide managers and leadership vital and 
unambiguous insights into an Sol in a timely manner better enabling the ability to deliver 


an Sol within budget and on time. 


The definition and components of MBSE as presented by Vaneman creates the 
basis for the understanding and implementation of MBSE within this thesis. In establishing 
this baseline for the definition and understanding of MBSE, the thesis has the ability to 


proceed with addressing the stated research questions. 


Cc. REVIEW OF DEPARTMENT OF DEFENSE DIGITAL 
ENGINEERING STRATEGY 

The DOD DE Strategy defines the purpose of integrating DE into DOD systems 
throughout their full life cycles, along with its vision of integration and five interconnected 
goals. Goal 1 specifies establishing formal methods to support the development, 
integration, and the universal employment of digital models as a fundamental element of 
engineering activities to provide a comprehensive digital representation of an Sol 
throughout its entire life cycle. The formal method will afford the ability to maintain 
consistent performance of analysis and decision making for programs throughout the DOD 
enterprise. Goal 1 is further decomposed into three separate, but interrelated, sub-goals: 


formalize the planning for models to support engineering activities and decision making 
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across the life cycle; formally develop, integrate, and curate models; and use models to 
support engineering activities and decision making across the life cycle (Office of the 


Deputy Assistant Secretary of Defense for Systems Engineering 2018). 


While MBSE does not account for all of aspects of DE, it is capable of capturing a 
significant portion of a system’s information and data. Through its application, MBSE 
enables the execution of a substantial portion of the DOD DE Strategy. One aspect of that 
application will require the employment of MBSE when performing SETR events. 
However, while the term MBSE and certain concepts date to 1993 and have continued to 
mature as computational power and technology have improved, MBSE still requires 
additional progress to develop a precise and consistent approach to modeling an Sol 


throughout their life cycles (International Council on Systems Engineering 2019). 


D. REVIEW OF DEFENSE ACQUISITION GUIDEBOOK: 
CHAPTER 3 

All DOD acquisition is fundamentally governed by two documents, DOD Directive 
5000.01 and DOD Instruction 5000.02. These documents provide guidance with the intent 
of ensuring the DOD procures quality products which are compliant to the stakeholders’ 
requirements providing measureable improvements to the specified capabilities necessary 
to perform the stated missions at a fair and reasonable price. While the documents were 
created in the 1970s, the DOD has updated them and added supplemental documents 
throughout the years. 


To compliment DOD Directive 5000.01 and DOD Instruction 5000.02 the DOD 
has published the Defense Acquisition Guidebook (DAG) as a resource for the acquisition 
workforce to provide best business practices which are tailorable for each program based 
on its needs and at the discretion of those performing the work. Each chapter of the DAG 
addresses a separate focus area that a program will perform at some point, or throughout, 
of the life cycle of its system. The DAG’s Chapter 3’s purpose is to provide guidance 
related to the general discipline of systems engineering by detailing a definition of systems 
engineering and its broad application, to include processes and efforts, within DOD 


acquisition programs. 


Chapter 3 of the DAG describes one of the most significant set of activities that 
occur during the development life cycle of an Sol within DOD acquisition programs. The 
System Engineering Technical Reviews (SETR) are a set of strategic assessments. These 
assessments permit a review board to interrogate the data and information of a program or 
Sol throughout its life cycle against a set of criteria tailored to each specific SETR. The 
review board investigates of a broad set of characteristics, based on the tailored criteria, 
regarding a program or Sol such as concept of operations, requirements, risks, schedule, 
program progress, technical maturity, manufacturability, and a host of other characteristics. 
Certain characteristics undergo review during particular phases of the DOD acquisition 
process within the systems’ development life cycle for a specific SETR, while other 
characteristics see review at every SETR. These activities result in vital periodic 
evaluations of the program or Sol that provide insight to its current health and status. The 
program’s or Sol’s level of technical maturity determines when the execution of any 
specific SETR occurs within its development life cycle. This policy ensures the occurrence 
of SETR’s are event-based and not schedule-based. Chapter 3 of the DAG states, 
“Technical Maturity Points identifies the objectives of each SE assessment and the 


technical maturity point marked by each review (Defense Acquistion University 2017).” 


Chapter 3 of the DAG provides the Technical Maturity Points within a table. 


Presented below is a modified version of that table which only denotes the SETR events. 


Table 1. . DOD Acquisition Technical Maturity Points. Adapted from 
Defense Acquistion University (2017, 43-47). 


Technical Maturity Points 





Technical Review Objective Technical Maturity Additional Information 
Point 
Recommendation Initial system 
that the preferred performance established 
‘ : : System parameters 

Alternative materiel solution : and plan for further 

: defined; balanced ? 
Systems Review | can affordably analyses (e.g., assessing 


with cost, schedule 


(ASR) meet user needs : technical maturity and 
: and risk. : ; 
with acceptable associated risks) supports 
risk. Milestone A criteria. 




















Technical Review 


Technical Maturity Points 


Objective 


Technical Maturity 


Additional Information 





System 
Requirements 
Review (SRR) 


System 
Functional 
Review (SFR) 


Capability 
Development 
Document (CDD) 
Validation 





technical configuration 


Recommendation 

to proceed into 
development with 
acceptable risk. 


Recommendation 
that functional 
baseline satisfies 
performance 
requirements and 
to begin 
preliminary 
design with 
acceptable risk. 


Requirements 
validation 
authority action. 
Provides a basis 
for preliminary 
design activities 
and the PDR. 





Point 


Level of 
understanding of top- 
level system/ 
performance 
requirements is 
adequate to support 
further requirements 
analysis and design 
activities. 


interdependent systems. 


Functional baseline 
established and under 
formal configuration 
control. System 
functions in the 
system performance 
specification 
decomposed and 
defined in 
specifications for 
lower level elements, 
that is, system 
segments and major 


subsystems. 


Major cost and 
performance trades 
have been completed 
and enough risk 
reduction has been 
completed to support 
a decision to commit 
to the set of 
requirements (1.e., 
CDD or equivalent) 





Government and 
contractor mutually 
understand system 
/performance 
requirements including: 
(1) the preferred materiel 
solution (including its 
support concept) from the 
Materiel Solution 
Analysis (MSA) phase; 
(2) plan for technology 
maturation; and (3) 
maturity of 


Functional requirements 
and verification methods 
support achievement of 
performance 
requirements. Acceptable 
technical risk of 
achieving allocated 
baseline. 


Support preparation for 
CDD validation by 
performing systems 

engineering trade-off 
analysis addressing 

relationships of cost, 
requirements, design, and 
schedule. Once validated, 

a Configuration Steering 

Board assumes 

responsibility to review 

all requirements changes 
and any significant 
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Technical Maturity Points 
Technical Maturity Additional Information 


Technical Review Objective 
Point 





changes for ACAT I and 
IA programs in 
development, production, 
and sustainment that have 
the potential to result in 
cost and schedule impacts 
to the program. 





Recommendation Allocated baseline 
established such that 


that allocated fe ee Ges 

baseline satisfies ic cone Preliminary design and 

38 : sufficient confidence ; : 
Preliminary user requirements : basic system architecture 
‘ : to proceed with ae 
Design Review and developer : : support capability need 
; detailed design. oP 
(PDR) ready to begin : and affordability goals 
Baseline also : 

and/or caps achievement. 


cette? | sapors 10 USC 
P 2366b certification, if 





sas applicable. 
Initial product baseline 
Recommendation established by the system 
to start detailed design 
Critical Design | . fabricating, Product design is documentation; 
Review (CDR) integrating, and _|stable. Initial product | affordability/should-cost 
testing test baseline established. goals confirmed. 


Government assumes 
control of initial product 
baseline as appropriate. 


articles with 
acceptable risk. 





Recommendation 
mah esy sienna Actual system (which 
tested has been ; 
verified We. represents the production 
Verification | Pout baseline Serer mano aie 
Review COMP Aan. my sleimeesien analysis jae 
(SVR)/Functional the functional _ \erified to conform to ene on endier ° 
; baseline) andis_ | functional baseline. : ° : 
Configuration pee testing. Synonymous with 
Audit (FCA) ali eA fan system-level Functional 
(operational Configuration Audit 
P (FCA). 


assessment) with 




















acceptable risk. 
Production Recommendation : : : ‘ 
: : Design and Production engineering 
peaullutess aa ecaeuen manufacturing are roblems resolved and 
Review (PRR) processes are g P 
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Technical Maturity Points 








Technical Review Objective Technical Maturity Additional Information 
Point 
mature enough to ready to begin ready to enter production 
begin limited production. phase. 
production with 
acceptable risk. 
Product baseline 
established. Verifies Ceitenationthatthe 
the design and 
; : system to be deployed 
Recommendation manufacturing 
; matches the product 
5 to start full-rate documentation, : 
Physical : : baseline. Product 
‘ production and/or | following update of . Boe tee 
Configuration : configuration finalized 
: full deployment | the product baseline : 
Audit (PCA) : and system meets user’s 
with acceptable to account for 
: needs. Conducted after 
risk. resolved OT&E : 
: OT&E issues are 
issues, matches the ie uadl 
physical : 
configuration. 
Manufacturing 
processes are mature 
Full-Rate and support full-rate 
Production Decision to begin sclera Delivers fully funded 
Decision Review full-rate d P y quantity of systems and 
. emonstrated in é : 
(FRP DR) or Full {production and/or nocaicnal supporting materiel and 
Deployment decision to begin bie einen services for the program 
Decision Review | full deployment. . or increment to the users. 
(FDDR) supporting full 
deployment (i.e., 
system validated 
through OT&E). 
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Chapter 3 of the DAG denotes the importance and value provided to both the 
government and developers by performing these reviews as follows: 

Department experience (e.g., Government Accountability Office (GAQ) 

Report 12—400SP) has found that successful programs use knowledge- 

based product development practices that include steps to gather knowledge 

to confirm the program’s technologies are mature, their designs are stable 

and their production processes are in control. Successful materiel 

developers ensure a high level of knowledge is achieved at key junctures in 

development. (Defense Acquistion University 2017, 43) 

SETR’s provide an opportunity to have insight into the health and status of a 
program. This insight can offer the possibility to detect and fix potential issues prior to 


allowing the continuation of development for an Sol where the cost of changes rises 


substantially the as the program progresses. 


E. REVIEW OF A ROADMAP FOR ADVANCING THE STATE OF 
THE PRACTICE OF MBSE FOR GOVERNMENT ACQUISITION 

The rate of MBSE adoption by prime contractors of government acquisition has 
increased in recent years. An increase in MBSE adoption has also begun within the 
government acquisition community. Noguchi states that the driving factor for prime 
contractors to increase MBSE adoption is the desire to improve “knowledge management, 
technical communication, data interchange, and increased efficiency” (Noguchi 2019). 
While for the government acquisition community, Noguchi asserts improved efficiency 
provides a driving force, but he also suggests a “need for a better approach for managing 
the complexity of those Government mission enterprises” (Noguchi 2019). Similar to the 
currently discontinuous MBSE definitions, the efforts to adopt MBSE, by both prime 
contractors and the government acquisition community, have been fragmented and often 
unconnected. The intent of Noguchi’s paper, as stated in its title: A Roadmap for Advancing 
the State of the Practice of MBSE for Government Acquisition, is to provide: 

...an initial attempt at establishing a shared vision to drive discussions 

among the Government acquisition community—and the associated 

community of prime contractors, systems engineering and integration 

(SE&I) contractors, and Federally Funded Research and Development 


Centers (FFRDC)—to drive investment and collaboration to improve the 
state of the practice of MBSE within this community. (Noguchi 2019, 1) 
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Noguchi presents the broad workings of MBSE along with a collection of benefits 
that MBSE is able to provide to a program regardless of the organization, governmental or 
not, that has chosen to employ MBSE. Additionally, he provides the structure of digital 
models within DE and their interactions across a generalized organizational hierarchy. This 
provided information establishes a base understanding of MBSE and its place within DE 
and a generalized organization to permit discussion of what Noguchi has stated as the four 
primary areas of MBSE that require focus for government application. The four primary 
areas of MBSE that require focus for government application are: 
|. Enabling Enterprise Systems Engineering 
2. Improving System Acquisition and Execution Outcomes 
3. Institutionalizing Evolved Systems Engineering 
4. Advancing the State of MBSE Tools (Noguchi 2019, 5) 
Although, Noguchi’s four focus areas contain similarities to the DOD DE 
Strategy’s five goals (presented below), significant differences exist between the two lists. 
Additionally, the DOD document addresses DE while Noguchi’s paper addresses MBSE. 
|. Formalize the development, integration and use of models to inform 
enterprise and program decision making 
Provide an enduring, authoritative source of truth 
3. Incorporate technological innovation to improve the engineering 
practice 

4. Establish a supporting infrastructure and environments to perform 
activities, collaborate, and communicate across stakeholders 

5. Transform the culture and workforce to adopt and support digital 


engineering across the life cycle (Office of the Deputy Assistant 
Secretary of Defense for Systems Engineering 2018, 4) 


Nm 


The DOD DE Strategy’s goals and focus areas are not exclusively sequential to 
their numbering; however, some of the tasking must naturally follow a sequential structure. 
As part of Goal 1 the DOD DE Strategy looks to first codify the development, integration 
and use of models to inform enterprise and program decision making into a unified practice 
regardless of the organization performing the work prior to full employment through the 
entire community of practice of government acquisition. After this formalization, the focus 
moves to multiple efforts to establish Goals 3 and 4 thereby, enabling the realization of 


Goals 2 and 5. 
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In contrast, Noguchi presents all of his focus areas as having concurrent efforts. 
Each of these focus areas have assigned efforts for the near term, longer term, and for their 
end state. In addition, each focus area has included generalized efforts performed as work 
to date. Another difference between the documents is that while Noguchi acknowledges 
MBSE requires improvement, the foundation of focus area 4, he advocates for a broad 
application of MBSE now rather than first formalizing development and integration of 


models. 


Despite the differences between the documents, many points of alignment exist. 
One such alignment is the agreement of the need to continue the establishment and 
refinement of MBSE application within SETRs in the near term. This required refinement 
was the impetus behind Vaneman, Carlson, and Wolfgeher paper (2020) and by extension 
this thesis. 


F. REVIEW OF DEFINING A MODEL-BASED SYSTEMS 
ENGINEERING APPROACH FOR MILESTONE TECHNICAL 
REVIEWS 


In support of achieving Goal 1 of the DOD DE Strategy, Vaneman; Carlson; and 
Wolfgeher performed research that culminated in the creation of a paper addressing a topic 
that literature has just begun to cover, MBSE’s integration into a critical aspect of DOD 
acquisition. Vaneman, Carlson, and Wolfgeher (2020) researched the three following 
questions: 

1. Define a systematic process for developing the virtual model of the 

system, as the program progresses through the acquisition life cycle. 

2. Identify representative SETR entrance criteria and related questions and 
determine how those questions could change to better represent the data 
required for system and program decisions in a MBSE environment. 

3. Define how the model of the system can be used in lieu of “artifacts” to 
provide decision-makers with a more complete representation of the 
system during SETRs. (3) 

In addition to supporting the DOD DE Strategy, the research also denotes a 

deficiency of the current SETR process well-known to DOD acquisition professionals, that 


the employment of a MBSE environment has the potential to solve. Vaneman, Carlson, 


and Wolfgeher (2020) state SETRs’ deficiency as: 
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Current SETRs are based around lengthy reviews of static, contractually 
obligated “artifacts” that are used to demonstrate successful completion of 
the entrance criteria. Participants typically ‘freeze’ these “artifacts” many 
days prior to the SETR in order to provide baselines from which to 
synchronize various products used during the review. This baselining and 
eventual loss of concordance between “artifacts” are the primary drawbacks 
when conducting reviews using “artifact-based” methods. (34) 


Vaneman, Carlson, and Wolfgeher detail how the generation of a systematic 
process to develop a virtual model of a system throughout its life cycle requires an iterative 
effort. As a system progresses thorough the five primary phases of the system acquisition 
life cycle the model expands to include additional information and requires updating of 
existing modeled data and relationships. However, since each element present within the 
system has a single representation within the virtual model of the system each element 


requires only a single update per iteration. 


Regarding the second research question, from which this thesis stems, Vaneman, 
Carlson, and Wolfgeher investigated if the current MBSE environment has the ability to 
evaluate comprehensively SETRs’ criteria, thereby enabling the execution of model-based 


SETRs. 


The integration of the MBSE environment into DOD acquisition has the possibility 
to allow model-based SETRs where the data reviewed against the criteria is the current 
system information. As previously discussed, system information within the MBSE 
environment, which focuses on the model, provides all observers and users of the system 
information a single continuous current understanding of the Sol. The model purpose is to 
capture not only the data inherent to the element modeled, but also its relationships to other 
elements. In the usage here, an element can denote anything from a system of systems to 


the least complex aspect of a system modeled. 


Based on their investigation of the second question Vaneman, Carlson, and 
Wolfgeher (2020) conclude that: 

MBSE supported early technical reviews, where traditional MBSE 

visualizations were very applicable. However, SETRs at PDR and later, 


showed less promise for being fully accomplished using current MBSE 
visualizations as evaluated today. (54) 
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While their investigation concluded at PDR, the presented conclusions provide 
guidance on where to focus future efforts. At least two primary areas that require focus are 
readily identifiable. First is the establishment of new MBSE visualizations based on those 
current SETR criteria that are currently fully un-addressable. The second, which this thesis 
will focus on, is an assessment of current PDR criteria to enable the generation of a process 
to curate SETR criteria and alter them as necessary. The process will allow a robust inquiry 
and thereby greater insight into the Sol under review. This will directly address the related 
findings of Vaneman, Carlson, and Wolfgeher that numerous PDR criteria are not suitable 
to be addressed within a MBSE environment. The criteria’s lack of robustness, often due 
to their binary nature, results in severely limited usefulness as part of reviewing the Sol. 


(Vaneman, Carlson, and Wolfgeher 2020) 


G. SUMMARY OF LITERATURE REVIEW 


Chapter II discussed various sources of literature regarding a spectrum of 
interconnected topics relating to MBSE. The chapter starts with a definition of MBSE and 
the related challenges in defining MBSE. The discussion progressed into a broad 
description of the DOD DE Strategy, which then quickly focused on its Goal | and a note 
that the application of MBSE will enable the execution of a significant portion of this 
strategy. In addition, the discussion mentioned one portion of the employment of MBSE 
would concentrate on MBSE’s role performing SETR events. To provide a common 
understanding the chapter provided a general account of DOD acquisition followed by an 
overview of SETR events to permit further investigation of the previous assertion. The 
examination of the assertion that a major portion of the DOD’s DE Strategy would involve 
MBSE’s employment in performing SETR events began with reviewing a proposed 
roadmap for the advancement of practicing MBSE within the federal government’s 
acquisition programs. The review compared and contrasted the roadmap to the DOD’s DE 
Strategy. A point of similarity between them lead to the chapter discussing research that 
reviewed and interrogated MBSE’s integration and ability to ingrate into SETR events. 
This research culminated in finding a number of deficiencies and shortcomings within both 


MBSE’s integration and the current SETR process. 


i 


Based on the presented significance and proven benefits of performing SETR it is 
the author’s assertion that being able to execute SETRs within a MBSE architectural 


framework is critical to successfully implementing the DOD DE Strategy. 
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Il. PROCESS 


A. PURPOSE 


As captured by one of the sub-goals within Goal 1 of the DOD DE Strategy, the 
creation of a formalized plan to enable the employment of models to support engineering 
activities and decision making across the life cycle is required. The process created as part 
of this thesis shall afford a tool employable within a formalized plan. The employment of 
the process on a set of SETR criteria allows for an assessment of three specific 
characteristics that each SETR criterion possess. The first characteristic assessed is the 
consistency with, or appropriateness to, the required level of development addressed by the 
SETR criterion in comparison to the current SETR event. The second characteristic is the 
SETR criterion’s quality to assess or its ability to provide insight into the health and status 
of the program or Sol under review. The final characteristic is each SETR criterion’s ability 
for representation by an existing MBSE view. These assessments will determine a set of 
SETR criteria ability to evaluate an Sol, to what extent criteria can map to existing MBSE 
views and the scope of criteria that require additional alteration either to the criteria or to 


MBSE views to permit the mapping of the criteria. 


The research performed as the basis of this thesis endeavored to employ the 
developed process to the PDR criteria previously collected and assessed by Vaneman, 
Carlson, and Wolfgeher (2020). The application of the process permits a verification of 
Vaneman, Carlson, and Wolfgeher’s findings and allows a better understanding of the state 
of the PDR criteria with regards the three characteristics. An additional outcome of 


performing the process is the quantification of the binary criteria. 


B. APPROACH 


The process detailed below provides an approach that permits a consistent and 
repeatable assessment of the three defined characteristics for each SETR criterion. 
Employment of this process within a formalized plan would support Goal 1 of the DOD 
DE Strategy. 
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1, Review, Document, and Alter SETR Criteria 


Review, Document, and Alter SETR Criteria is the title of Function 0.0. The title 
of the top-level function captures the intent of the process as a whole. Function 1.0 through 
Function 3.0 sequentially assesses one of three characteristics for each SETR criterion as 
specified within the function’s name. The results of the assessments, within each function, 
determines the next step for each criterion assessed. As necessary, the criterion will 
undergo alteration to enhance the criterion’s ability to enable a robust evaluation of the 
program or Sol thereby providing insight into the current health and status, while 
maintaining (if appropriate) the original intent. Each function that performs assessments 
also documents the results of the related assessments. In addition, the documentations 
includes any alteration or rewording performed on the criteria. Finally, Function 4.0 acts 
as a measure against separate lists of documented results and rewording of criteria. It 
accomplishes this by producing one single document that captures a consolidation of all 
the documented assessments of the SETR criteria. As seen within Figure 1, the LML action 
diagram Review, Document, and Alter SETR Criteria decomposes into four lower level 


functions. 
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Figure 1. Review, Document, and Alter SETR Criteria LML Action 
Diagram 
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2. Assess and Document Each Current SETR Event Criterion’s Level of 
Development 


Function 1.0 decomposes into five lower level functions as shown in Figure 2. 
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Figure 2. Assess and Document Each Current SETR Event Criterion’s Level 
of Development LML Action Diagram 





The first lower level function, Function 1.1, is the first assessment performed on 
every SETR criterion reviewed. The assessment evaluates whether the criterion matches 
the documented required or expected level of development for an Sol at the appropriate 
level of development for the current SETR event. The possible responses are ‘YES’, 


‘PARTIAL’, and ‘NO’. 


The employment of the DAG permitted the determination of each criterion’s 
appropriateness in relation to the required or expected level of development for a current 
SETR event. The DAG consists of multiple chapters that denote what is generally required 
for each SETR event. However, it is not possible for the DAG to predict every potential 
program or Sol that may undergo review. To accommodate this the DAG states to tailor 
each SETR based on the program or Sol under review (Defense Acquistion University 


2017). This is where the systems engineer must exercise exceptional teamwork and 
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communication to enable all relevant stakeholders and domain specialist the ability to 


provide meaningful input in tailoring the SETR criteria. 


A response of ‘YES’ indicates no further assessment of this characteristic for this 
criterion is required and the criterion progresses onto Function 1.2. Function 1.2 documents 
all criteria assessed as containing a level of development consistent with the appropriate 
level of development for the current SETR event as well as the justification for the 
assessment. No further actions occur within Function 1.0 regarding these criteria 


documented within Function 1.2. 


A response of ‘PARTIAL’ indicates additional assessment is required for this 
criterion. Function 1.3 documents all criteria assessed as containing a level of development 
either partially consistent or inconsistent with the appropriate level of development for the 
current SETR event. Rather than only documenting these criteria, it is critical to document 
the justification of the assessment. In documenting the justification, the reasoning and any 
objective quality evidence is available for questioning and as an example for future 


assessments. 


A criterion assessed as containing a level of development partially consistent with 
relation to the current SETR event will progress to Function 1.4. Each criterion that 
advances to Function 1.4 will undergo a further assessment. Figure 3 shows the LML action 
diagram of Assess and Document Each Partially Consistent SETR Criterion Ability to be 
Reworded to Allow Consistency with Level of Development for Current SETR Event, 


which consists of five lower level functions. 
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Figure 3. | Assess and Document Each Partially Consistent SETR Criterion 
Ability to be Reworded to Allow Consistency with Level of Development 
for Current SETR Event LML Action Diagram 


The first lower level function, Function 1.4.1, will assess whether rewording the 
criterion would afford the ability to alter its question such that it addresses a subject that is 
consistent with the appropriate level of development for the current SETR event, while 


maintaining the criterion’s original intent. The possible responses are ‘YES’ or ‘NO’. 


To determine whether rewording these partially consistent criteria would afford the 
ability to change their questions a thorough examination of each criterion’s question and 
the likely response(s) is required. The utilization of the DAG, other acquisition resources, 
along with communication between all relevant stakeholders and domain specialist to 
provide meaningful input in tailoring the SETR criteria are vital to making an accurate 


assessment. 


A response of ‘NO’ indicates no further assessment of this criterion will occur 
beyond documentation of the criterion’s result as well as the justification for the 
assessment. In documenting the justification, the reasoning and any objective quality 


evidence is available for questioning and as an example for future assessments. 


A response of “YES’ indicates if reworded the criterion has the ability to design a 


question that is consistent with the appropriate level of development for the current SETR 
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event while maintaining the criterion’s original intent. Each criterion assessed as such 
progresses to documentation of the assessment, rewording of the criterion, and finally a 
documentation of the reworded criterion. Function 1.4.3 performs the documentation of all 
criteria assessed with the necessary ability for rewording. In addition to documenting the 
result of the assessment, there is benefit in documenting the justification for the assessment. 
After documentation, Function 1.4.4 performs a rewording of the criterion thereby 
designing a question that addresses a subject consistent with the appropriate level of 


development for the current SETR event, while mindful of the original intent. 


When rewording the criterion one should consider whether the rewording makes 
the criterion: binary in nature, have no or partial ability to provide insight into the health 
and status of the Sol, or prevents its mapping to an existing MBSE view. While these are 
not desirable characteristics, the intent of the Function 1.4.4 does not expressly address 
altering the criterion regarding those characteristics. However, even though these 
characteristics will undergo an assessment later there are benefits to addressing these 
characteristics during this rewording as it will reduce the amount of effort required when 
performing additional assessments later in the process. Ultimately, the person performing 
this function must ensure the reworded criterion will enable the ability to design a question 
that addresses a subject that is consistent with the appropriate level of development for the 


current SETR event. 


Function 1.4.5 is the final lower function of Function 1.4. The function documents 
all criteria reworded in Function 1.4.4. Capturing what each reworded criterion now states 


is vital to ensuring its continued assessment throughout the remainder of the process. 


Function 1.5 is the final lower level function within Function 1.0. This function 
takes note of the criteria deemed as inconsistent during the assessment performed within 
Function 1.1 and the partially consistent criteria deemed as unfit for rewording as part of 
the assessment performed within Function 1.4.1. Function 1.5 then removes these criteria 
from the pool of SETR criteria that will undergo further assessment within Function 2.0. 
However, they will undergo collection and consolidation with any other removed criteria 


within Function 4.0. 
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3. Assess and Document Each SETR Criterion’s Ability to Provide 
Insight into the Health and Status of the Sol 


A criterion assessed to have a level of development consistent with the appropriate 
level of development for the current SETR event, whether initially or after rewording, will 
progress to Function 2.0. Each criterion that advances to Function 2.0 will undergo 
assessment and documentation regarding the criterion’s quality to assess an Sol or its 
ability to provide insight into the health and status of the Sol under review. Figure 4 shows 
the LML action diagram of Assess and Document Each SETR Criterion’s Ability to 
Provide Insight into the Health & Status of the Sol, which consists of five lower level 


functions. 
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Figure 4. Assess and Document Each SETR Criterion’s Ability to Provide 
Insight into the Health & Status of the Sol LML Action Diagram 


The first lower level function is Function 2.1 that performs an assessment of each 
criterion regarding whether it can provide insight into the health and status of the Sol under 
review. The possible responses are ‘YES’, ‘PARTIAL’, and ‘NO’. To determine whether 
each criterion can provide insight into the health and status of the Sol under review, 
examination of both the nature of the question and the likely response is required. One 
particular check performed for every criterion relates to whether the question is binary in 


nature. If the criterion is binary in nature the best one can hope for is to achieve limited 
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insight, based on a likely response of yes or no, without further questioning that may not 
occur. A criterion assessed as binary in nature would result in a ‘PARTIAL’ response, 
unless additional aspects of the criterion prevented the ability to provide insight. In the 
situation stated above where the criterion has a binary nature and another aspect prevents 


the ability to provide insight, assessment would result in a response of ‘NO’. 


A response of ‘YES’ indicates no further assessment of this criterion for this 
characteristic is required. The following is one example of a reviewed PDR criterion 
deemed to have a ‘YES’ response for this assessment. “What are the inter-service 
maintenance requirements, organic and contractor mix, projected workloads, installation 
requirements and time phasing for accomplishing depot maintenance requirements?” This 
criterion is non-binary in nature and inquires about required critical aspects of a system 
under development. A criterion assessed to have a response of “YES’ progresses to 
Function 2.2. It is within this function that the documentation of all similarly assessed 
criteria occurs. Along with documenting, the criteria deemed to have the ability to provide 
insight into the health and status of the Sol under review; benefits exist in additionally 
documenting the justification for the assessment. No further actions within Function 2.0 


occur regarding the criteria documented within Function 2.2. 


A response of ‘PARTIAL’ indicates additional assessment is required for this 
characteristic of this criterion. Function 2.3 will document all criteria assessed as 
containing a partial ability to provide insight into the health and status of the Sol under 
review as well as the justification of each assessment. In documenting the justification, the 
reasoning and any objective quality evidence is available for questioning and as an example 


for future assessments. 


After the documentation within Function 2.3, the criteria assessed to have a partial 
ability to provide insight into the health and status of the Sol will progress to Function 2.4. 
Here each criterion will undergo assessment and documentation regarding its ability to 
provide insight into the health and status of the Sol under review if reworded. Figure 5 
shows the LML action diagram of Assess and Document SETR Criteria with Partial Ability 
to Provide Insight into the Health and Status of the Sol if Reworded, which consists of five 


lower level functions. 
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Figure 5. Assess and Document SETR Criteria with Partial Ability to 
Provide Insight into the Health and Status of the Sol if Reworded LML 
Function Diagram 


Function 2.4.1 performs an assessment of each criterion regarding whether if 
reworded could it provide insight into the health and status of the Sol under review. If the 
previous assessment was partial, the question within the criterion may have a binary nature 
or another aspect preventing the criterion’s ability to provide insight. These criteria 
previously assessed as partial will likely present a minimal effort in rewording to provide 
the required ability. In regards to binary criterion, simple changes to how one poses a 
question permits the removal of the binary nature of the criterion while maintaining the 
intent of the question. Some examples of this could change asking ‘if a particular task has 
completed’ to ‘how the particular task was completed’, ‘when it was completed’, ‘by what 
date will it be completed’. Each of these potential changes removes the binary nature of 
the criterion and provides the ability to increase the level of insight into the health and 
status of the Sol under review. However, any criteria assessed as having a partial ability to 
provide insight that are not binary in nature may present more of a challenge when 
rewording. To determine whether these criteria have the ability to provide insight into the 
health and status of the Sol under review after rewording, examination of the criterion’s 
question and the likely response is required. Ultimately, the possible responses for this 


assessment regarding rewording of the criteria are “YES’ or ‘NO’. 
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A response of ‘NO’ indicates this criterion undergoes documentation within 
Function 2.4.2. This function documents these criteria as well as the justification for the 
assessment. In documenting the justification, the reasoning and any objective quality 


evidence is available for questioning and as an example for future assessments. 


Criterion assessed as having a response ‘YES’ progresses to documentation of the 
assessment, rewording of the criterion, and finally documentation of reworded criterion. 
Function 2.4.3 performs the first of these actions with the documentation of all criteria 
assessed as having the ability to provide insight into the health and status of the Sol under 
review if reworded. Along with documenting the criteria as deemed to have the ability to 
provide insight into the health and status of the Sol under review if reworded, there is 
benefit in documenting the justification for the assessment. After this documentation, 
Function 2.4.4 performs a rewording of these criteria. While mindful of the original intent 
and maintaining it as appropriate, this function rewords the criterion thereby bestowing it 


the ability to provide insight into the health and status of the Sol under review. 


While not required at this point, as assessment of the other reviewed characteristics 
of criteria that have progressed through this function will occur later, when rewording the 
criterion one should consider whether the rewording prevents the criterion for mapping to 
an existing MBSE view. If possible, addressing this characteristic during this rewording 
will reduce the amount of effort required for these criteria later in the process. Ultimately, 
the person performing this function must ensure the reworded criterion will enable the 


ability to provide insight into the health and status of the Sol under review. 


Function 2.4.5 is the final lower level function within Function 2.4. The function 
documents all criteria reworded in the Function 2.4.4. Certain criterion documented herein 
could exist with the following characteristic, a partial or no capability of mapping to an 
existing MBSE view. Although, this is not a desirable characteristic the intent of Function 
2.4.4 did not expressly address altering the criterion regarding that characteristic. Capturing 
what each reworded criterion now states is vital to ensuring its continued assessment 


throughout the remainder of the process. 
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Function 2.5 is the final lower level function within Function 2.0. This function 
compiles the criteria deemed as having no ability to provide insight during the assessment 
performed within Function 2.1 and the criteria with a partial ability to provide insight 
deemed not capable of rewording as part of the assessment performed within Function 
2.4.1. Function 2.5 takes this compiled list of criteria and removes them from the pool of 
SETR criteria that will undergo further assessment within Function 3.0. They will undergo 


collection and consolidation with any other removed criteria within Function 4.0. 


4. Assess and Document SETR Criteria Ability to be Mapped to an 
Existing MBSE View 


A criterion assessed to have the ability to provide insight into the health and status 
of the Sol under review, either initially or after rewording, will progress to this function. 
Each criterion that advances to Function 3.0 will undergo assessment for its ability to map 
to an existing MBSE view. Figure 6 shows the LML action diagram of Assess and 
Document SETR Criteria Ability to be Mapped to an Existing MBSE View, which consists 


of three lower level functions. 
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Figure 6. Assess and Document SETR Criteria Ability to be Mapped to an 
Existing MBSE View LML Action Diagram 
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The first lower level function within Function 3.0 is Function 3.1. This function 
performs of an assessment of each criterion regarding whether the criterion is capable of 
mapping to an existing MBSE view. The possible responses are ‘YES’, ‘PARTIAL’, and 
‘NO’. To determine whether each criterion is capable of mapping to an existing MBSE 
view, the person performing the assessment must have a robust understanding of the 
application of systems engineering diagram, models, and data employed. This knowledge 
will enable the identification of potential MBSE views relevant to the criterion evaluated 
thereby allowing the mapping of the information or data required to address the criterion 


onto a MBSE view. 


A response of “YES’ or ‘PARTIAL’ indicates the data or information required to 
address the criterion is partially or fully able to map to an existing MBSE view. The 
following criterion is an example with a response of ‘YES’. “Has an allocated baseline, or 
equivalent, been established, and is it complete and under configuration control?” The 
existing MBSE view Operational Activity to System Traceability Matrix or SV-7 has the 
ability to capture and represent the information required to address this criterion. Each 
criterion assessed as either response progresses to Function 3.2. This function documents 
all criteria assessed as having a partial or full capability to map to an existing MBSE view. 
Certain criterion documented herein could exist with only a partial capability of mapping 
to an existing MBSE view. These criteria require capture, as they are able to provide some 
insight into the system of interest. These captured criteria that provide partial insight 
provide value through an awareness of what MBSE views require potential creation or 
alteration in the near future. Along with documenting the criteria as capable of mapping to 
an existing MBSE view, there is benefit in additionally documenting the justification for 
the assessment. No further actions occur to the criteria documented within Function 3.2 


within Function 3.0. 


A response of ‘NO’ indicates the data or information required to address the 
criterion is unable to map to an existing MBSE view. The reason for this assessment varied 
extensively. Two of the primary reasons included either information that MBSE does not 
address or a MBSE view that could address does not currently exist. The following is one 
example of information that MBSE does not address. “Was a systems engineering 
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knowledgeable chairperson assigned?” These criteria will progress to Function 3.3. This 
function documents all criteria assessed as not capable of mapping to an existing MBSE 
view. Rather than only documenting the criteria as deemed not currently capable of 
mapping to an existing MBSE view, the function shall also document the justification of 
each assessment. In documenting the justification, the reasoning and any objective quality 
evidence is available for questioning and as an example for future assessments. No further 


actions occur to the criteria documented within Function 3.3 within Function 3.0. 


5. Consolidate Documentation of All Assessed SETR Criteria 


Throughout the process, documentation of the results from the assessments of each 
criterion’s characteristics has occurred. As a measure against this disparate set of 
documentation leading to a loss of data, a consolidation of all the documentation capturing 
the assessments and rewording of the SETR criteria will occur to produce one single 


document. 


Function 4.0 consists of three lower level functions. Figure 8 shows the LML action 


diagram of Consolidate Documentation of All Assessed SETR Criteria. 





Figure 7. Consolidate Documentation of All Assessed SETR Criteria LML 
Action Diagram 
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Function 4.1 is the first lower level function within Function 4.0. This function 
assembles and documents the consolidation of all criteria assessed to have a resulting ‘NO’ 
or ‘NO / PARTIAL’ response as part of any evaluation performed by any function within 
the process. This assembled list has the possibility to provide various insights to the criteria 
and the Sol such as areas of weakness, areas with insufficient consideration, or areas for 


growth. 


The second lower function, Function 4.2, collects and documents the consolidation 
of all criteria assessed to have a resulting “YES’ or ‘YES / PARTIAL’ response as part of 
any evaluation performed by any function within the process response. This collected list 
has the potential to provide various insights to the criteria and the Sol such as areas of 


strength, areas over considered/managed/resourced, or areas for opportunity. 


Function 4.3 is the final lower level function within Function 4.0. This function 
merges and documents this consolidation of all criteria consolidated within Function 4.1 
and Function 4.2. In addition, all reworded criterion will undergo collection and 


consolidation for the purpose of documentation within this Function. 


The document created herein this Function will act as the single source of 
documentation regarding the review, documentation, and alteration of the SETR criteria. 
This list will empower leadership in decision making regarding the employment or 
exclusion of criteria well before SETR events, to enable improved evaluations of Sols. 
Additionally, it will provide insight into what level of MBSE integration is achievable with 
the existing MBSE views and the SETR criteria. 


a2 


IV. DATA ANALYSIS 


A. ISSUE RESEARCHED AND DATA ANALYZED 


Previous research investigated the implementation of integrating MBSE into SETR 
events. This research concluded that at PDR, and subsequent SETR events, the vast 
majority of criteria were unable to map onto existing MBSE views thereby preventing 
representation of the criteria and its related information and data with the current set of 
MBSE views. As part of the findings it was documented that a preponderance of the PDR 
criteria were binary in nature and thereby provided little to no insight into the “health and 
status” of the Sol. These PDR criteria, which are binary in nature, do not afford information 
or data that is suitable to robust employment within a MBSE environment (Vaneman, 


Carlson, and Wolfgeher 2020). 


The research performed in support of this thesis analyzed the PDR criteria by 
evaluating three primary characteristics of each criterion. The first characteristic governs 
the criterion’s consistency with, or appropriateness to, the required or expected level of 
development for an Sol under review during PDR. The second characteristic relates to the 
criterion’s quality to assess an Sol or its ability to provide insight into the health and status 
of the Sol under review. The final characteristic addresses whether each criterion has the 


capability to map to an existing MBSE view. 


When performing the criteria assessments, the research did not distinguish between 
the criteria area of focus (e.g., programmatic or technical). The research only assessed each 
criterion based on the three previously mentioned characteristics. Six criteria of the 753 
criteria did not undergo assessment based on lack of information or understanding of the 
content within the criteria. The supplemental Microsoft Excel document captures the 


NAVAIR PDR criteria analyzed and the results of the research. 
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B. RESEARCH RESULTS 


1. Results of Function 1.0: Assessment of Characteristic 1 


The results of performing Function 1.1 to assess each criterion’s first characteristic 
revealed 420 of the 747 criteria, or nearly 56.4% of all assessed criteria, were determined 
consistent with the required or expected level of development for an Sol under review 
during the preliminary design phase. The following is an example of a criterion deemed 
appropriate or consistent to evaluate an Sol at the preliminary design level of development. 


“Has a System Functional Review (SFR) been successfully completed?” 


The evaluation of the criteria comprised the employment of various chapters from 
the DAG along with additional acquisition instructions, policies, guidance, and 
documentation. While it is not possible for the DAG to predict every potential program or 
Sol that may undergo review it does provide a broad scope of expected criteria based on 
the SETR event and general structure of the Sol (e.g., hardware intensive, defense unique 
software intensive, incrementally fielded software intensive, accelerated acquisition, 
hardware dominant hybrid, software dominant hybrid). The remaining 327 criteria 
consisted of criteria determined either partially consistent (195 or nearly 26% of all 
assessed criteria) or inconsistent (132 or nearly 17.7% of all assessed criteria). Figure 8 


details the distribution of the assessed criteria. 
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Figure 8. Distribution of Assessed Criteria Regarding Their Consistency to 
an Expected Level of Development for an Sol during the Preliminary 
Design Phase 


Criteria assessed as partially consistent primarily contained one of two reoccurring 
traits. The first trait related to criteria addressing multiple deliverables or artifacts of which 
some were not appropriate for a preliminary design level of development. An example is 
numerous criteria made inquiries regarding the system’s Capability Development 
Document (CDD) and its Capability Production Document (CPD). A validated CDD is 
required as an entry criterion for PDR making this portion of those criteria appropriate. 
However, the creation of a CPD will not occur until after the Critical Design Review (CDR) 
and likely near the end of the Engineering and Manufacturing Development (EMD) Phase 
thereby making this portion inconsistent. The results of this example lead to the assessment 
of those criteria as partially consistent. The other reoccurring trait involved criteria that 
inquired about outcomes regarding deliverables that would exist only in a draft status at 
the time of PDR. As partial information or data would exist at the time of PDR the 
assessment of partially consistent occurred for the criteria with this trait. The EMD Request 


for Proposal (RFP) and the Life Cycle Sustainment Plan are examples of artifacts that at 


35 


the time of PDR would exist in a draft status. In the case of the EMD RFP it may have 
released, but still under source selection, preventing any data regarding a contractor’s 


response or information. 


Criteria assessed as inconsistent predominantly contained one trait. The trait 
consisted of inquiring about either a specific deliverable or group of deliverables not yet 
required or available at the time of PDR. Each of these criteria with this trait appeared 
improperly placed within PDR and belonged within an alternative SETR event that more 
closely aligned to the level of development addressed within each of these criteria. One 
example is numerous criteria made inquiries regarding elements of the system regarding 
the system’s CPD. The creation of a CPD does not occur until after the CDR and likely 
near the end of the EMD Phase thereby making these criteria inconsistent. Another 
example of this trait, numerous criteria addressed elements of the system in regards to and 
traits of Performance Based Logistics (PBL). However, per DAG Chapter 4 (Defense 
Acquistion University 2019) and the PBL Guidebook (Assistant Secretary of Defense 
Logistics & Material Readiness 2016) state that the transition to PBL shall occur no sooner 


than post Milestone C. 


The performance of Function 1.4 included the further assessment of the criteria 
deemed partially consistent. This further assessment interrogated whether rewording of the 
criterion would permit the criterion consistency with the required or expected level of 
development for an Sol under review during PDR. The further assessment of the partially 
consistent criteria resulted in 194 of the 195 criteria deemed as possessing the ability for 
rewording that allowed consistency. Rewording the criteria to remove the deliverable(s) 
inconsistent with a PDR level of development allowed those specific criteria to become 
consistent. Table 2 shows examples of criteria deemed partially consistent as currently 
written within NAVAIR’s PDR criteria and a possible way to reword them to attain 
consistency. While the assessment performed at this point of the process only addresses 
the criterion’s first characteristic, as suggested within Chapter 3 the rewording also looks 
to remove other undesirable traits such as a binary nature. The supplemental Microsoft 


Excel document captures the justification for the assessment of Characteristic 1 for each 
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criterion. The justification details the reason for the assessment, often with a source sited, 


that provides direction regarding how to reword the criterion. 


Table 2. 


Assessment of 


Rewording 


Criterion as Currently Written 


Examples of Criteria as Written Contrasted with Suggested 


Criterion Suggested Rewording 





(Oi keteveoutiacen | 


Partially 
Consistent 


Partially 
Consistent 








Does the AS address a plan to 
satisfy Human Systems 
Integration (HSI) requirements 
for each domain addressed in the 
Capability Development 
Document (CDD) / Capability 
Production Document (CPD), 
including minimum standards for 
those domains not specifically 
addressed in the CDD / CPD? 


Have facilities / test resources 
(contractor and Government) been 
defined and included in the 
planning? 





How does the AS plan to 
address satisfying Human 
Systems Integration (HSI) 
requirements for each domain 
addressed in the Capability 
Development Document 
(CDD), including minimum 
standards for those domains not 
specifically addressed in the 
CDD? 
What are the — required 
Government facilities and test 
resources defined and included 
in the planning? What required 
contractor facilities and test 
resources does the EMD RPF 
capture? What facilities and test 
resource planning for T&E in 
later acquisition phases has 
taken place? 








As part of performing Function 1.4.3, the 133 criteria (one from partially consistent 


criteria set and the 132 inconsistent criteria set) deemed inconsistent or unfit for rewording 


underwent documentation and then removal from the set of criteria for all future 


assessments. The following is the partially consistent criterion deemed unfit for rewording. 


“Transient support requirements when the system requires some level of support for 


continental U.S. and outside continental U.S. activities that are not regular homeports or 


support sites.” Deciphering the intent of the criterion could not occur, as this criterion does 


not pose a question or is it even a complete sentence. It is unclear if whether this derives 
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from a transcription error of the original criterion or if the verbiage presented captures the 


actual criterion. 


The partially consistent criteria deemed suitable for rewording to address a level of 


development appropriate to a preliminary design numbered 194. 


The combination of the criteria deemed consistent initially (420 criteria or nearly 
56.4% of all assessed criteria) with the partially consistent criteria deemed consistent after 
rewording (194 criteria, approximately 26% of all assessed criteria) created a revised list 


of 614 criteria for assessment of the second characteristic. 


2. Results of Function 2.0: Assessment of Characteristic 2 


The performance of Function 2.1 consisted an assessment of the second 
characteristic to determine whether each criterion can provide insight into the health and 
status of the Sol. This assessment reviewed the revised list of 614 combined criteria. The 
results revealed that only 19 criteria, or approximately 3% of the 614 criteria reviewed, can 
provide insight into the health and status of the Sol. The following criterion is one example 
deemed to have the ability to provide insight. “What is the status versus the critical path?” 
The results also assessed 558 criteria, or approximately 91%, to have a partial ability to 
provide insight into the health and status of the Sol. The remaining 36 criteria, or 
approximately 6%, had no ability to provide insight into the health and status of the Sol. 
Figure 9 displays the distribution of the assessed 614 criteria. 
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@ Criteria with no ability to 
provide insight into health & 
status of Sol 


@ Criteria with partial ability to 


provide insight into health & 
status of Sol 
@ Criteria with ability to 
558, 90.88% provide insight into health & 
status of Sol 





Figure 9. Distribution of Assessed Criteria Regarding Their Ability to 
Provide Insight into the Health and Status of the Sol 


The assessment of the second characteristic involved the examination of both the 
nature of the question and the likely response. The review identified two traits of the criteria 
that attributed to the high percentage of criteria deemed as having partial or no ability to 
provide insight into the health and status of the Sol. The first trait, which was the 
predominate trait, governed whether the criterion under review had a binary nature and 
thereby limited their potential to provide insight into the health and status of the Sol. When 
a criterion is binary in nature the best one can hope for is to achieve limited insight, based 
on a likely response of yes or no, without further questioning that may not occur. An 
example of a binary in nature criterion with partial ability to provide insight is “Has the 


software schedule been updated based upon the preliminary software design?” 


The second trait related to whether the criterion under review duplicated another 
criterion within the list of criteria. The duplication covered both word for word duplications 
and those criteria that duplicated the intent of another criterion. While this trait occurred 
much less frequently than the first trait, their inclusion is detrimental to the SETR event. 


39 


The following criterion is an example of a binary in nature criterion that is also duplicative 
and as such, one of the two entries of the criterion has no ability to provide insight. “Have 
developmental test plans been formulated in accordance with the TEMP?” Two instances 
of this criterion exist word for word within the set of NAVAIR PDR criteria; with the only 


variation between the two criteria based in the defining of TEMP within one criterion. 


The performance of an additional assessment occurred, as part of the performance 
of Function 2.4, on those criteria determined to have partial ability to provide insight into 
the health and status of the Sol. The further assessment of the 558 criteria, deemed to have 
partial ability to provide insight, resulted in the determination that 553 of the criteria 
possess the capability for rewording. The criteria deemed not to have the ability for 
rewording consisted of criteria assessed as duplicative. Figure 10 displays the distribution 
of the results from the further assessment of the 558 criteria deemed to have a partial ability 


to provide insight into the health and status of the Sol. 


5, 0.90% 


@ Partial Ability Criteria that 
have the ability for 
rewording to bestow ability 
to provide insight into health 
& status of Sol 

@ Partial Ability Criteria that 
have no ability for rewording! 
to bestow ability to provide 
insight into health & status 
of Sol 


















Figure 10. Distribution of Assessment Results of Criteria with Partial Ability 
to Provide Insight Regarding Their Ability for Rewording to Bestow the 
Ability to Provide Insight 


40 


Table 3 shows examples of criteria assessed with partial ability to provide insight 
as currently written within NAVAIR’s PDR criteria and a possible way of how to reword 


them to provide insight. 


Table 3. Examples of Criteria as Written Contrasted with Possible 
Rewording 


Assessment of 
Characteristic 2 


Criterion as Currently Written Criterion Possible Rewording 





What long lead items been 
identified? How are they being 
addressed? How mature are the 
production processes? What are 

the indications they are 
sufficiently mature for this 
phase of the program? 
What software requirements 
have been allocated to COTS, 

GOTS and reused software? 

How was it determined the 

allocated requirements are 


appropriate? 


Have long lead items been 
Partial Ability to | identified, and are the production 
Provide Insight | processes sufficiently mature for 
this phase of the program? 





Are software requirements 
allocated to COTS, GOTS and 
reused software appropriate? 


Partial Ability to 
Provide Insight 














Documenting the results of the assessment is the first half of performing Function 
2.4.2. Removing the 41 criteria (five from the set of criteria with partial ability and the 36 
criteria with no ability to provide insight) with either no ability to provide insight or 
unsuitable for rewording, from the set of criteria for all future assessments, is the second 


half of Function 2.4.2. 


After the assessment of the second characteristic, a second merger of the criteria 
occurred to create a further revised list of criteria for the assessment of the third 
characteristic. The revised list comprised 573 criteria from the initially assessed to have 
(20 criteria) the ability to provide insight or able to attain the ability to provide insight 
through rewording (553 criteria). 
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3. Results of Function 3.0: Assessment of Characteristic 3 


Finally, the 573 criteria from the revised list of PDR criteria underwent an 
assessment, performed as part of Function 3.0, to evaluate each criterion’s third 
characteristic. The third criteria characteristic regards each criterion’s ability to map to an 
existing MBSE view. The results from this review closely aligned with the findings of 
Vaneman, Carlson, and Wolfgeher (2020). The assessment found only 75 criteria, or 
approximately 13% assessed of having some ability to map to an existing MBSE. Two 
groups of criteria existed within the 75 criteria. The first consisted of 60 criteria that had 
the ability to map to an existing MBSE view. The second group contained of 15 criteria 
that had a partial ability to map to an existing MBSE view. Figure 11 displays the 
distribution of the 573 assessed criteria regarding their ability to map to an existing MBSE 


view. 


60, 10.47% 


@ Criteria with no ability to map 
to existing MBSE view 


@ Criteria with partial ability to 
map to existing MBSE view 


@ Criteria with ability to map to 
existing MBSE view 


498, 86.91% 





Figure 11. Distribution of Assessed Criteria Regarding Their Ability to map 
to an Existing MBSE View 


The 75 criteria with some ability to map to existing MBSE views originated from 


27 thematic areas. Thirteen of these 27 thematic areas assessed to have the most promise 
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of MBSE satisfying their criteria based on the criteria ability to map to existing MBSE 
views. Table 4 documents the number of criteria that were deemed partially or fully able 
to map to existing MBSE views, along with the total number of criteria within each of the 


Thematic Areas. 


Table 4. Distribution of Criteria within Most Promising Thematic Areas 























Number of NESE of 
aes (Oyirour) Total 
Criteria Partiall Number 
Thematic Area | Mappable to an M bl y f MBSE Views 
Existing MBSE /2Ppabletoan oo 
5 Existing MBSE __ Criteria 
View § 
View 
; . CV-2, CV-3, 
aa a 8 : 16 | OV-4, OV-5a, 
OV-5b, PV-3 
Cost / Schedule / 
Performance / Key ; 1 > SV-7 
Performance 
Parameters (KPP) 
Human Systems SV-1, SV-5a, 
Integration > - 19 
; ; SV-5b, 
Engineering 
Net-Centric 
Operations and 
Warfare 
Consolidated 2 : ° Sade aae 
Compliance 
Checklist 
Net-Centric 
Operations and 5 ; 5 SV-5a, SV- 
Warfare Policy 5b, StdV-1 
Requirements 
Performance 4 : 9 CV-6, SV-Sa, 
Requirements SV-5b, SV-7 
Work 
Breakdown 
Production Costs : 1 > Structure, 
Estimates Cost 
Breakdown 
Structure 
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Number of 
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Senos Partiall Number 
Thematic Area Mappable to an y MBSE Views 
Existing MBSE Mappable to an of 
: Existing MBSE Criteria 
View ; 
View 
oe i) - 9 Risk Matrix 
Assessment 
: SV-1, SV-3, 
Requirements 8 ; 18 SV-5a, SV- 
Management 5b 
Safety and Health 3 - 4 StdV-1 
System Constraints 
(Budgets) 7 2 9 SV-1, SV-7 
System Preliminary 5 ; 5 SV-3. SV-4 
Design : 
System 
Requirement 3 ; 6 SV-5a, SV- 
Assessment for 5b, SV-7 
Individual Cl’s 




















‘: INTERPRETATIONS OF RESULTS 


The results of this research show that after rewording the set of PDR criteria would 
decrease in size, from 753 to 573 criteria; however, the remaining criteria would address a 
level of development consistent with the expected or required level of development for 
PDR. Furthermore, the criteria would have the ability to provide insight into the health and 
status of the Sol through asking robust, pointed, and meaningful questions. Additionally, 
the results display that the bulk of the PDR criteria are unable to support MBSE integration 


when employing existing MBSE views. 


A conclusion one can draw from the data is the effectiveness of any PDR performed 
with this set of criteria lacks the intended robustness and usefulness when attempting to 


provide insight into the health and status of the Sol under review. 


The data from the research shows that over 40% of the criteria assessed are partially 
or fully inconsistent with evaluating an Sol at a preliminary design level of development. 
In addition, the data finds that 589 of 614 criteria deemed consistent, whether initially or 
after rewording, as having a binary nature. In other words, nearly 96% of the consistent 
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criteria contain a binary nature, thereby limiting their potential to provide insight into the 
health and status of the Sol. The implication is all SETR criteria likely require some level 


of curation unless the reviewed PDR criteria are severe outliers. 


Taking a deeper look at the data of the remaining 573 criteria after the first two 
assessments, additional observations are possible along with associated potential 
implications. One can see the distribution of the criteria; regarding whether the criteria 
require rewording and which characteristics the criteria did not possess thereby 
necessitating rewording. Figure 12 details this distribution of the 573 criteria that remain 


after the first two assessments. 






377, 65.79% 


176, 30.72% 


@ Criteria assessed as not requiring rewording to consistency or the abiltiy to provide insight 
® Criteria assessed as only requiring rewording to address consistency 


@ Criteria assessed as only requiring rewording to address the abiltiy to provide insight 








@ Criteria assessed as requiring rewording to address both consistency and abiltiy to provide insight 


Figure 12. Distribution of Remaining Criteria After the First Two 
Assessments Detailing the Characteristics Requiring Rewording 


An observation from this deeper look is that without rewording only 19 criteria or 
2.54% of all PDR criteria assessed are consistent with the required or expected level of 
development and have the ability to provide insight into the health and status of the Sol. 


As seen within Figure 6, this increases to 3.32% after removing criteria either inconsistent, 
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no ability to provide insight, or capable of rewording to enable either of these 
characteristics. However, after rewording the revised list of assessed PDR criteria 573 (or 
approximately 76.7%), are consistent with the required or expected level of development 
and have the ability to provide insight into the health and status of the Sol. This observation 
shows curation of the PDR criteria is a mandatory step to ensure proper evaluation of 
programs and Sols. The curation should include, at least, an assessment of the level of 
development each criterion addresses; whether each criterion is binary in nature; and if any 


duplication of the criteria exists prior to attempting MBSE integration into SETR events. 


The observation made during the review of the previously collected PDR criteria 
relates to the terminology employed. Numerous criteria and PDR criteria categories or 
thematic areas all employed terminology no longer in use within the acquisition life cycle 
or DOD acquisition in general. The System Development and Demonstration (SDD) phase 
is one example that occurred frequently; however, it is not part of the acquisition life cycle. 
Having outdated terminology introduces confusion and uncertainty for both reviewers and 
personnel attempting to provide information thereby reducing the overall efficacious of the 


SETR event. 
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Vv. CONCLUSIONS 


A. SUMMARY 


In support of executing Goal 1 of the DOD digital engineering Strategy, the 
research performed as part of this thesis created a process to assess and permit curation of 
the SETR criteria. The assessments within the process looked to curate the SETR criteria, 
but also provide a more complete understanding of the SETR criteria health and status with 
respect to appropriateness to the SETR event, quality of assessing the Sol, and ability to 
map to an existing MBSE view. In determining the health and status of the SETR criteria 
one has the ability to comprehend the likely quality of any SETR event performed with 
those specific criteria. Additionally, the third portion of the assessment affords the insight 
into the readiness of the SETR criteria for integration with MBSE through use of existing 
MBSE views. The curation, coupled with the knowledge provided, allows the ability to 
improve or enhance the existing SETR criteria for use within MBSE and for SETR events 


in general. 


The previously discussed questions and their responses, captured below, shaped the 


original intent of the process and the realization of the final process. 


1. How does one determine if a process to curate existing SETR criteria and 
alter them as necessary to more accurately capture the “health and status” 


of an Sol under review through model-based views and data is required? 


The execution of an assessment of the SETR’s criteria critical characteristics 
provided insight into the health and status of the criteria. This insight supplied the 
knowledge of whether the SETR criteria would have the ability to accomplish their 
intended purpose. The results of the assessment presented a bleak reality of the SETR 
criteria reviewed. Over 43% of the criteria did not fully address the appropriate level of 
development as aligned with the SETR event. The criteria that proceeded to the next 
assessment had nearly all criteria, approximately 97%, partially or fully failing to meet the 
evaluation requirement of having the ability to provide insight into the health and status of 


the Sol. The final assessment of the remaining criteria found only 75 criteria, or 


47 


approximately 13% assessed, had the ability to map partially or fully to an existing MBSE 


view. 


Ze How does one create a process to curate existing SETR criteria and alter 
them as necessary to more accurately capture the “health and status” of an 


Sol under review through model-based views and data? 


Based on the status of the SETR criteria, the process created, and presented within 
Chapter 3, ultimately focused on assessing and curating the existing SETR criteria. The 
assessment entailed the review of SETR criteria characteristics and alteration as necessary 
to more accurately capture the “health and status” of an Sol under review. The process also 


addressed assessing the SETR criteria’s ability to map to an existing MBSE view. 


This process supports one of the three sub-goals of Goal 1 of the DOD digital 
engineering Strategy. The sub-goal is to formalize the planning for models to support 
engineering activities and decision making across the life cycle. The process formalizes a 
way to curate SETR criteria in preparation of MBSE integration. Additionally, the process 
denotes which criteria can map to existing MBSE views thereby providing a gauge of the 


current readiness of specific SETR criteria to have MBSE integrated. 


a: How does one verify that a solution generated to address the problem 


fulfills its intended purpose? 


The research performed on the previously gathered NAVAIR PDR criteria and the 
resulting outcome provide verification that the process can detect inadequacies within an 
existing set of SETR criteria. The use of these newly curated criteria in the performance of 


a PDR could provide validation. 


B. RECOMMENDATIONS FOR FUTURE WORK AND 
CONSIDERATION 


The following four recommendations capture potential future work, which would 


continue supporting the execution the DOD digital engineering Strategy’s Goal 1. 


The first recommendation based on the state of the reviewed SETR criteria is that 


additional SETR criteria undergo assessment and curation in preparation for MBSE 
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integration. Having a healthy and robust set of SETR criteria that can accurately capture 
the “health and status” of an Sol under review is necessary prior to the integration and 


implementation of MBSE into SETR events. 


The second recommendation is for the acquisition community to assign necessary 
attributes to SETR criteria during their development and curation. The performance of this 
effort would mirror what systems engineering teaches its practitioners regarding the 
assignment of necessary attributes for requirements during their development. While the 
requirements’ attributes may not all directly translate to necessary or desirable attributes 
for SETR criteria, they are a good starting point. Working from one of the existing set of 
requirements’ attributes, or a combination of sets, a group of SETR criteria attributes could 
undergo identification with the goal of a codified set of attributes. Even with the 
understanding that SETR criteria perform a separate role than requirements, ensuring the 
application of systems engineering rigor to the generation and curation of criteria will lead 


to improved evaluation, and thereby understanding, of the Sol. 


The third recommendation is to investigate the creation of additional MBSE views 
that will support the evaluation of Sols. While the created methodology assessed each 
criterion’s ability to map to an existing MBSE view, the requirement for additional MBSE 
views will exist for the full integration of MBSE into SETR events. A need exists to 
perform an evaluation regarding the needed MBSE views to support a curated set of SETR 


criteria. 


The final recommendation is to consider the automation of the applicable portions 
of SETR criteria assessments. The curation of SETR criteria has inherent assessments that 
only a human can perform. However, the performance of some assessments could leverage 
Formal Methods and software programing. The employment of Formal Methods in 
conjunction with the development of a software program could directly handle some 
evaluations while other assessments could have their burden reduced through the assistance 
the software provides by decreasing the initial workload prior to human interaction of 


assessing SETR criteria. 
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SUPPLEMENTAL: NAVAIR PDR CRITERIA AND RESULTS OF 
ASSESSMENT 


This supplemental Microsoft Excel document captures a large portion of the 
research performed in support of this thesis. It may be obtained through the Dudley Knox 
Library of the Naval Postgraduate School. 


This portion of the research comprised of an evaluation consisting of three separate 
assessments of each PDR criterion for three specific characteristics that each should 
possess. The first characteristic assessed is the consistency with, or appropriateness to, the 
required level of development addressed by the SETR criterion in comparison to the current 
SETR event. The second characteristic is the SETR criterion’s quality to assess or its ability 
to provide insight into the health and status of the program or Sol under review. The final 
characteristic is each SETR criterion’s ability for representation by an existing MBSE 
view. The assessments determine a set of SETR criteria ability to evaluate an Sol, to what 
extent criteria can map to existing MBSE views, and the scope of criteria that require 
additional alteration either to the criteria or to MBSE views to permit the mapping of the 
criteria. The document contains all of the NAVAIR PDR criteria analyzed along with the 


results and the associated explanations for each assessment. 


51 


THIS PAGE INTENTIONALLY LEFT BLANK 


52 


LIST OF REFERENCES 


Estefan, Jeff A. 2008. Survey of Model-Based Systems Engineering (MBSE) 
Methodologies. INCOSE MBSE Initiative, INCOSE. 


Department of Defense. 2009. DOD Architecture Framework Version 2.0. DoDAF V2.0. 
Washington, DC: Department of Defense, May 28. 


Dam, Steven H. 2014. DOD Architecture Framework 2.02 - A Guide to Applying System 
Engineering to Develop Itegrated, Executable Architectures. Manassas: SPEC 
Innovations. 


Assistant Secretary of Defense Logistics & Material Readiness. 2016. PBL Guidebook. 
Guidebook, Department of Defense. 


Jackson, Maddelena, Marcus Wilkerson, and Jean-Francois Castet. 2016. Exposing 
hidden parts of the SE process: MBSE patterns and tools for tracking and 
traceability. 2016 IEEE Aerospace Conference. Big Sky: IEEE. 1-12. 


Vaneman, Warren K. 2018. Evolving Model-Based System Engineering Ontologies and 
Structures. 28th Annual INCOSE International Symposium. INCOSE. 1-10. 


Defense Acquistion University. 2017. “Defense Acquisition Guidebook Page Viewer: 
Chapter 3 Systems Engineering.” Defense Acquisition University. September 25. 
Accessed March 09, 2019. https://www.dau.edu/tools/dag/Pages/DAG-Page- 
Viewer.aspx?source=https://www.dau.edu/guidebooks/Shared%20Documents %2 


OHTML/Chapter%203 %20S ystems %20Engineering.aspx. 








2019. Chapter 4 Life Cycle Sustainment.pdf. February 27. Accessed 2020. 


https://www.dau.edu/guidebooks/_layouts/15/WopiFrame.aspx?sourcedoc=/guide 
books/Shared%20Documents/Chapter% 204 %20Life%20Cycle%20Sustainment.p 


df&action=default. 





Office of the Deputy Assistant Secretary of Defense for Systems Engineering. 2018. 
Department of Defense Digital Engineering Strategy. Washington, DC: 
Department of Defense. 


Goldwasser, Debora Arena, and Christopher Ryder. 2019. Model-Based Systems 
Engineering as the Catalyst for a Rapid Acquisition Process (RAP). 2019 IEEE 
International Systems Conference (SysCon). Orlando: IEEE. 1-7. 


Holladay, Jon B., Jessica Knizhnik, Karen J. Weiland, Amanda Stein, Terry Sanders, and 
Paul Schwindt. 2019. MBSE Infusion and Modernization Initiative (MIAMI): 
“Hot” Benefits for Real NASA Applications. 2019 IEEE Aerospace Conference. 
Big Sky: IEEE. 1-14. 


ae 


International Council on Systems Engineering. 2019. Transitioning Systems Engineering 
to a Model-based Discipline -SEBoK. October 31. Accessed May 11, 2020. 


https://www.sebokwiki.org/wiki/Transitioning Systems Engineering to_a_ Mode 
1-based_Discipline. 





Noguchi, Ryan A. 2019. A Roadmap for Advancing the State fo the Practice of Model 
Based Systems Engineering for Government Acquistion. INCOSE International 
Symposium. Orlando: INCOSE. 678-690. 


Vaneman, Warren K., Jerry J. Sellers, and Steven H. Dam. 2018. Essential LML — Life 
cycle Modeling Language (LML): a Thinking Tool for Capturing, Connecting and 
Communicating Complex Systems. Manassas, VA: SPEC Innovations. 


Vaneman, Warren K, and Ronald Carlson. 2019. Model-Based Systems Engineering 
Implementation Considerations. Proceedings of the 13th Annual IEEE Systems 
Conference. Orlando: Institute of Electrical and Electronics Engineers. 





. 2020. “Evaluating Current Systems Engineering Models for Applicability to 
Model-Based Systems Engineering Technical Reviews.” Naval Engineers Journal 
1-8. 


Vaneman, Warren K., Ronald R. Carlson, and Christopher Wolfgeher. 2020. Defining a 
Model-Based Systems Engineering Approach for Milestone Technical Reviews. 
Acquistion Research Program Sponsored Report Series, Monterey, CA: Naval 
Postgraduate School. 


54 


INITIAL DISTRIBUTION LIST 


Defense Technical Information Center 
Ft. Belvoir, Virginia 


Dudley Knox Library 


Naval Postgraduate School 
Monterey, California 


55 


